10 - 704 : Information Processing and Learning Fall 2016 Lecture 6 : Sept 19

1. Integral Estimate: Ĥ(x) = − ∫ p̂(x) log p̂(x)dx 2. Re-substitution Estimate: Ĥ(x) = − 1 n ∑n i=1 log p̂(Xi) where p̂ is obtained using the samples {X1, ..., Xn}. 3. Splitting Data Estimate: Ĥ(x) = − 1 m ∑m i=1 log p̂(Xi) where p̂ is obtained using the samples {Xm+1, ..., Xn}. A cross-validation estimate can be defined similarly, e.g. the leave-one-out estimate is given as Ĥ(x) = − 1 n ∑n i=1 log p̂i(Xi) where p̂i is obtained using all samples except Xi.